chore(testing): add test discovery mode for pytest by gnufede · Pull Request #18559 · DataDog/dd-trace-py

gnufede · 2026-06-10T11:21:20Z

What

Adds a test discovery mode to the ddtrace/testing pytest plugin. When DD_TEST_OPTIMIZATION_DISCOVERY_ENABLED=true, pytest collects all tests and writes them as JSON Lines to a file, then exits without running any test. This is used by ddtest during its plan phase to enumerate the test suite before deciding which tests to run or skip.

Why

ddtest needs a language-agnostic way to enumerate tests for a given project. For Ruby, datadog-ci-rb provides this via a discovery mode. This PR adds the equivalent for pytest, producing the same JSONL format so ddtest's planner can parse both without special-casing the language.

Configuration

Env var	Default	Description
`DD_TEST_OPTIMIZATION_DISCOVERY_ENABLED`	`false`	Enable discovery mode
`DD_TEST_OPTIMIZATION_DISCOVERY_FILE`	`.testoptimization/tests-discovery/tests.json`	Output file path

Output format

Each line is a JSON object with:

name — test name
suite — test suite (file path, dot-separated)
module — module (directory, dot-separated)
parameters — JSON-encoded parameters for parametrised tests, or null
suiteSourceFile — relative path to the source file

Tests decorated with pytest.mark.skip or pytest.mark.skipif with a truthy non-string condition are excluded from the output.

Other changes

Moves _get_test_parameters_json and _encode_test_parameter from plugin.py to utils.py to avoid a circular import with _discovery.py
Registers DD_TEST_OPTIMIZATION_DISCOVERY_ENABLED and DD_TEST_OPTIMIZATION_DISCOVERY_FILE in the ddtrace settings registry
CI Visibility initialisation is suppressed when discovery mode is active (no spans emitted)

Add DD_CI_TEST_DISCOVERY_MODE_ENABLED support to the ddtrace/testing pytest plugin, matching the format produced by datadog-ci-rb. When enabled, pytest collects tests and writes them as JSON Lines to DD_CI_TEST_DISCOVERY_OUTPUT_PATH (default: ddtest/test_discovery/tests.json), then exits without running any test. Each line contains name, suite, module, parameters, and suiteSourceFile fields. Tests decorated with pytest.mark.skip or pytest.mark.skipif with a truthy non-string condition are excluded from the output. CI Visibility initialisation is suppressed when discovery mode is active. Also moves _get_test_parameters_json and _encode_test_parameter from plugin.py to utils.py so they can be imported by _discovery.py without a circular dependency. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

datadog-official · 2026-06-10T11:21:35Z

Tests

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 8 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64, 1]

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741491-d2b8243-musllinux_1_2_x86_64, 1]

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741357-d2b8243-manylinux2014_aarch64, 1]

View all 8 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 7 jobs - 7 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: d76a31e | Docs | Datadog PR Page | Give us feedback!}

…ery module Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cit-pr-commenter-54b7da · 2026-06-10T11:27:14Z

Codeowners resolved as

ddtrace/internal/settings/_supported_configurations.py                  @DataDog/apm-sdk-capabilities-python @DataDog/apm-python
ddtrace/testing/internal/constants.py                                   @DataDog/ci-app-libraries
ddtrace/testing/internal/pytest/_discovery.py                           @DataDog/ci-app-libraries
ddtrace/testing/internal/pytest/plugin.py                               @DataDog/ci-app-libraries
ddtrace/testing/internal/pytest/utils.py                                @DataDog/ci-app-libraries
supported-configurations.json                                           @DataDog/apm-sdk-capabilities-python @DataDog/apm-python
tests/testing/internal/pytest/test_plugin.py                            @DataDog/ci-app-libraries
tests/testing/internal/pytest/test_pytest_discovery.py                  @DataDog/ci-app-libraries

Use DD_TEST_OPTIMIZATION_DISCOVERY_ENABLED and DD_TEST_OPTIMIZATION_DISCOVERY_FILE (ddtest's canonical names) instead of the Ruby-aligned DD_CI_TEST_DISCOVERY_MODE_ENABLED / DD_CI_TEST_DISCOVERY_OUTPUT_PATH. ddtest owns the shared env var contract; individual language plugins should conform to it. Also update the default output path to match ddtest's .testoptimization/tests-discovery/tests.json. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…output The module field must match what the plugin reports at run time so the backend can correlate discovery results with test runs via FQN (module.suite.name). Using the static string "pytest" as module would collapse every test across all repos into the same namespace. nodeid_to_names already extracts the correct value (e.g. "tests.unit" for tests/unit/test_foo.py); we just needed to stop discarding it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…y mode Register TestOptHooks specs during discovery so item_to_test_ref can call the same custom hook chain used at run time. This means name/suite/ module resolution stays in sync with ITR matching if the logic ever changes, rather than being a separate parallel implementation. pytest-bdd is intentionally not wired up: BddTestOptPlugin is not registered in discovery mode, so BDD tests fall back to nodeid-based names (same as before). A TODO marks where to add that support. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add DD_TEST_OPTIMIZATION_DISCOVERY_ENABLED and DD_TEST_OPTIMIZATION_DISCOVERY_FILE to supported-configurations.json and regenerate _supported_configurations.py so the CI check passes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…est plugin - Sort discovery imports alphabetically into the ddtrace.testing block - Drop the pytest_collection_finish module-level re-export (caused F811 clash with BddTestOptPlugin.pytest_collection_finish); register the _discovery module directly as a plugin in pytest_configure instead - Fix import order and remove spurious f-string prefix in test_pytest_discovery.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

pytester.monkeypatch was added in pytest 6.2; older envs (py3.9 CI) don't have it. Accept monkeypatch as a separate fixture parameter. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 12ec7caea9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

… skip in discovery @pytest.mark.skipif with no condition argument yields condition=None in _is_item_skipped. Previously this fell through to continue, so such tests were included in discovery output even though pytest treats missing conditions as unconditional skip (matching plugin.py's _get_skipif_condition). Align the None branch with plugin.py: condition is None → return True. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…s_item_skipped pytest >= 7 raises a collection error for @pytest.mark.skipif with no condition argument, so such items never reach _is_item_skipped. Treating condition=None as an unconditional skip would diverge from pytest's own behaviour and handle a case that can't occur in practice. Restore the original: condition=None conservatively includes the test, same as a string condition that cannot be evaluated at collection time. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…y included Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…arg) pytest 9 accepts @pytest.mark.skipif(reason=...) without a condition and skips the test at runtime, but condition is still None at collection time. The conservative-include behaviour is correct; update the comment to reflect the actual pytest 9 behaviour rather than claiming it can't arise. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Move DD_TEST_OPTIMIZATION_DISCOVERY_ENABLED/FILE constants to ddtrace.testing.internal.constants alongside other DD_TEST_OPTIMIZATION_* constants to centralize config loading - Move get_workspace_path import to module level (no circular dependency) - Fix Windows path bug: use .as_posix() for consistent forward slashes - Open output file with explicit encoding="utf-8" - Remove unused _encode_test_parameter re-export from plugin.py; update test_plugin.py to import directly from utils.py - Add test for empty collection (output file created but empty) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(ci_visibility): use env.get() instead of os.environ.get in discov…

7f52620

…ery module Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

gnufede and others added 3 commits June 10, 2026 11:28

gnufede added changelog/no-changelog A changelog entry is not required for this PR. CI App labels Jun 10, 2026

gnufede and others added 4 commits June 10, 2026 11:52

fix(testing): suppress mypy misc error on pytest.hookimpl decorator

70b4b0a

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(testing): use monkeypatch fixture instead of pytester.monkeypatch

05c9977

pytester.monkeypatch was added in pytest 6.2; older envs (py3.9 CI) don't have it. Accept monkeypatch as a separate fixture parameter. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

gnufede changed the title ~~chore(ci_visibility): add test discovery mode for pytest~~ feat(ci_visibility): add test discovery mode for pytest Jun 11, 2026

gnufede changed the title ~~feat(ci_visibility): add test discovery mode for pytest~~ chore(testing): add test discovery mode for pytest Jun 11, 2026

Merge branch 'main' into gnufede/ddtest-discovery-format

12ec7ca

gnufede marked this pull request as ready for review June 11, 2026 15:01

gnufede requested review from a team as code owners June 11, 2026 15:01

gnufede requested review from mabdinur and wconti27 June 11, 2026 15:01

chatgpt-codex-connector Bot reviewed Jun 11, 2026

View reviewed changes

Comment thread ddtrace/testing/internal/pytest/_discovery.py

gnufede and others added 4 commits June 11, 2026 15:15

docs(ci_visibility): comment why condition=None/str are conservativel…

986172d

…y included Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

mabdinur approved these changes Jun 11, 2026

View reviewed changes

gnufede and others added 2 commits June 11, 2026 17:26

chore(ci_visibility): fix import sort order in test_plugin.py

b2b56f3

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

anmarchenko approved these changes Jun 12, 2026

View reviewed changes

Merge branch 'main' into gnufede/ddtest-discovery-format

d76a31e

gh-worker-dd-mergequeue-cf854d Bot merged commit 6d1992a into main Jun 12, 2026
508 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the gnufede/ddtest-discovery-format branch June 12, 2026 12:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(testing): add test discovery mode for pytest#18559

chore(testing): add test discovery mode for pytest#18559
gh-worker-dd-mergequeue-cf854d[bot] merged 17 commits into
mainfrom
gnufede/ddtest-discovery-format

gnufede commented Jun 10, 2026 •

edited

Loading

Uh oh!

datadog-official Bot commented Jun 10, 2026 •

edited

Loading

Uh oh!

cit-pr-commenter-54b7da Bot commented Jun 10, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

gnufede commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Configuration

Output format

Other changes

Uh oh!

datadog-official Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

cit-pr-commenter-54b7da Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codeowners resolved as

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gnufede commented Jun 10, 2026 •

edited

Loading

datadog-official Bot commented Jun 10, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jun 10, 2026 •

edited

Loading